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Abstract 

Background: The concept of early health technology assessment, discussed well over a decade, has now been 
collaboratively implemented by industry, government, and academia to select and expedite the development of emerging 
technologies that may address the needs of patients and health systems. Early economic evaluation is essential to assess 
the value of emerging technologies, but empirical data to inform the current practice of early evaluation is limited. We 
propose a systematic review of early economic evaluation studies in order to better understand the current practice. 

Methods/design: This protocol describes a systematic review of economic evaluation studies of regulated health 
technologies in which the evaluation is conducted prior to regulatory approval and when the technology effectiveness is 
not well established. Included studies must report an economic evaluation, defined as the comparative analysis of 
alternatives with respect to their associated costs and health consequences, and must evaluate some regulated health 
technology such as pharmaceuticals, biologies, high-risk medical devices, or biomarkers. We will conduct the literature 
search on multiple databases, including MEDLINE, EMBASE, the Centre for Reviews and Dissemination Databases, and 
EconLit. Additional citations will be identified via scanning reference lists and author searching. We suspect that 
many early economic evaluation studies are unpublished, especially those conducted for internal use only. 
Additionally, we use a chain-referral sampling approach to identify authors of unpublished studies who work 
in technology discovery and development, starting out with our contact lists and authors who published relevant 
studies. Citation screening and full-text review will be conducted by pairs of reviewers. Abstracted data will include 
those related to the decision context and decision problem of the early evaluation, evaluation methods (e.g., data 
sources, methods, and assumptions used to identify, measure, and value the likely effectiveness and the costs and 
consequences of the new technology, handling of uncertainty), and whether the study results adequately address 
the main study question or objective. Data will be summarized overall and stratified by publication status. 

Discussion: This study is timely to inform early economic evaluation practice, given the international trend in early 
health technology assessment initiatives. 



Background 

Economic evaluation is the comparative analysis of alter- 
native technologies with respect to their costs and con- 
sequences [1]. It is often used late in the evaluation of 
drugs, devices, and other technologies to inform cover- 
age decisions [2]. This is problematic because the evi- 
dence portfolio at the time of the reimbursement 



* Correspondence: ba.pham@theta.utoronto.ca 

^Toronto Health Economics and Technology Assessment Collaborative, 

University of Toronto, Leslie Dan Pharmacy Building, 6th floor. Room 651, 

144 College Street, Toronto, Ontario M5S 3M2, Canada 

^Institute of Health Policy, Management and Evaluation, University of 

Toronto, 155 College St, Toronto, Ontario M5S 3M2, Canada 

Full list of author information is available at the end of the article 

(3 BioMed Central 



decision is often incomplete. Cost and preference data 
are often not gathered prospectively within clinical trials 
designed to evaluate effectiveness. This means that ana- 
lysts cannot directly use trial data and must struggle to 
assemble evidence from a variety of sources and settings 
to evaluate the cost-effectiveness of new interventions 
[3]. Published data suggest that considerable economic 
uncertainty existed in about half of the submissions for 
reimbursement decisions [4]. Also, clinical trial design is 
rarely informed by ex ante economic modeling. This 
means that optimal strategies and the full range of sub- 
groups may not have been fully considered prior to com- 
mencing the clinical research program. 
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Economic evaluation can be used early in the process 
of technology evaluation. As an example, Vallejo-Torres 
et al. illustrate a three-stage cost-effectiveness analysis of 
a new medical device: an early phase in which simple 
methods are used to estimate the maximum price attain- 
able for the technology, a mid-stage which synthesizes 
data into a cost-effective model and identifies which in- 
formation is most valuable to reduce decision uncer- 
tainty, and a late stage in which all relevant information 
is synthesized [5]. The results suggest that early and it- 
erative economic evaluation could be useful to inform 
decisions along the technology development process. 

Early evaluation can contribute to decision-making by 
both industry and government [6,7]. From an industry's 
perspective, early evaluation may be used for early mar- 
ket assessment, managing research and development 
portfolios, and informing pricing and reimbursement 
scenarios. From the policy perspective, decision makers 
may benefit from information supplied by early evalu- 
ation. Thus, there is interest from both innovators and 
payers in early health technology assessment and, more 
specifically, early economic evaluation, to inform plan- 
ning and development decisions by industry and to in- 
form the potential of new technologies that may meet 
health system needs [8]. 

Though interest in early economic evaluation is con- 
siderable, much applied research is still at the pilot stage. 
There is uncertainty about which technologies can bene- 
fit from early evaluation, what methods are appropriate, 
and the contribution of early economic evaluation to 
health technology assessment [8]. The aim of this sys- 
tematic review is to describe the characteristics of early 
economic evaluation of emerging technologies and to 
understand current methods to early evaluation. 

Methods/design 

This protocol describes a systematic review of economic 
evaluation studies that are conducted to inform the de- 
velopment and planning of early evaluation of emerging 
health technologies. 

Inclusion criteria 

An economic evaluation is defined as the comparative 
analysis of alternatives with respect to their associated 
costs and health consequences. A study is included if it 
reports an early economic evaluation of a regulated 
health technology in order to inform the development of 
the technology (e.g., pre-clinical research or phase I or II 
studies). For this review, regulated health technologies 
include pharmaceuticals, biologies, medical devices, or 
biomarkers. The term 'biologic' refers to a class of thera- 
peutics that are produced by means of biological pro- 
cesses, including, for example, the blood production- 
stimulating protein erythropoietin, a growth- stimulating 



hormone, a gene therapy, a monoclonal antibody, or a 
vaccine [9]. The term 'medical device' refers to a class II 
device (e.g., blood pressure monitors, contact lenses, 
pregnancy test kits, single-use surgical instruments, 
catheters), a class III device (e.g., ventilators, cardiac 
monitors, hip implants, knee implants, lasers, chlamydia 
test kits, glucose meters), or a class IV device (e.g., defi- 
brillators, pacemakers, coronary stents, HIV test kits, 
neurosurgical shunts) that requires product licensing for 
general marketing purposes (examples are from a pres- 
entation by Ms. Sarah Chandler of Health Canada to 
MaRS -Excite, September 2012, with permission). The 
term 'biomarker' (or a biological marker) refers to an ob- 
jectively measured and evaluated indicator of a normal 
biological process, a pathogenic process (e.g., a DNA se- 
quence that is associated with susceptibility to disease), 
or a pharmacologic response to a therapeutic interven- 
tion [10]. 

Literature search 

The literature search will be conducted on multiple da- 
tabases, including MEDLINE (Ovid), MEDLINE In- 
Process (Ovid), EMBASE (Ovid), the Centre for Reviews 
and Dissemination: National Health Services Economic 
Evaluation Database (NHS FED), the Cochrane Library: 
Health Technology Assessment Database (HTA Data- 
base), and EconLit. The first four sources are standard 
repositories of economic evaluation studies. The last 
database is a reliable source of citations and abstracts of 
economic research [11]. All databases will be searched 
from the date of inception until present. 

The search used both Medical Subject Headings 
(MeSH) as well as keyword variants of all relevant terms, 
including keyword terms derived from MeSH scope 
notes to increase retrieval of citations from Medline In- 
Process. The search developed in MEDLINE will be 
adopted with the syntax and subject headings appropri- 
ate for the other databases mentioned. In addition to 
early economic evaluation keywords and MeSH terms 
generated through scoping search (below), the final 
search strategy will include the validated NHS EED 
search filter and a published MEDLINE G and EMBASE 
G filter to identify potentially relevant citations of eco- 
nomic evaluation studies in MEDLINE and EMBASE 
[12]. Both MEDLINE G and EMBASE G filters were 
tested and showed that they improve the search preci- 
sion [12]; however, both have been modified to conform 
to this study's needs. Namely, the subject heading for 
randomized control studies (RCTs) was removed from 
original filters, since we did not want to limit the re- 
trieved studies only to RCTs. Limits that are part of 
NHS EED, MEDLINE G, and EMBASE G filters in- 
cluded humans and publication type. The final search 
was not limited by language. 
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Since the early economic evaluation concept is not 
well defined and it is not included in the MeSH con- 
trolled vocabulary, an initial scoping search was per- 
formed in order to identify potential candidate keyword 
terms as well as MeSH subject headings. A set of seed 
articles was used to develop the MeSH terms and key- 
words used in the literature search [2,5-8,13-22]. These 
articles were selected because they are commonly cited 
and highly relevant to early health technology assess- 
ment or early economic evaluation. The relevant MeSH 



terms harvested from the seed articles include 'device 
approval,' 'diffusion of innovation,' and 'decision support 
techniques.' 

From the seed articles, we extracted the cited key- 
words and other keywords that are deemed relevant to 
the aspects of early economic evaluation. Table 1 de- 
scribes these aspects and the associated keywords. The 
first aspect is that the evaluation is conducted at an early 
phase in the development of the emerging technology. 
The objective of an early evaluation is to examine the 



Table 1 Aspects of early economic evaluation and associated keywords used in the literature search 



Aspects of early economic evaluation 



Keywords 



The objective is to examine tine (lil<ely) cost-effectiveness of tine 
emerging teclinology 



Tine evaluation is to provide decision support for different stal<eliolders 
involved in the development and diffusion of the emerging technology 



Limited supporting evidence for efficacy or effectiveness of the 
emerging technology 



Early economic evaluation has an influence on the design and future 
performance of the emerging technology 



Evaluation is conducted at a very early or early stage in the 
development of the emerging technology 



(Like$ OR potential).tw. adj2 (effect$ OR safety OR cost-effect$ OR efficiency 
OR impact).tw. 

headroom.tw. OR (effect$ adj2 gap).tw. 

(earl$ OR iterativeS OR continuS OR ongoing OR 'on-going' OR dynamicS 
OR exSante OR pre$develop$ OR constructive).tw. adj'3 (econom$ OR 
'health economS' OR pharmacoeconom$ OR analys$ OR evaluation OR 
frames OR approachS OR assessments OR Bayesian OR 'health technology 
assessment' OR 'technology assessment').tw. 

'development-accompanying'.tw. 

('earlySstage' OR earlS).tw. adj2 decision?.tw. 

(priority adj2 set?).tw. OR decision.tw adj2 (gateS OR pointS).tw. 

(products adj2 developmentS).tw. OR (policS adj2 developmentS).tw. 

('in-licensS' OR 'in licensS' OR inlicensS).tw. 

'only in research'.tw. OR 'only with research'.tw. OR 'in the context of 
research'.tw. OR (coverS adj2 evidence).tw. 

((interim OR conditions OR ((dependent OR dependant OR contingent) adj 
(upon OR on)) OR restricts) adj2 (fundS OR financed OR reimburses OR 
covers OR approvS OR listS OR accessS OR licensS OR licencS)).tw. OR 
'conditions of coverageS'.tw. OR 'conditions of reimburseS'.tw. OR (pricing 
or commercialization)^ 

((investment? or marketing or promotS or progressS or advancement) adj2 
(decisions or incentives or inducements or uncertaintS or incertitudes or 
economicS)).tw. 

'go no-go'.tw. OR 'go no go'.tw. OR 'go\no-go'.tw. OR 'go\no go'.tw. OR 
'go \ no-go'.tw. OR 'go \ no go'.tw. 

((earlS OR scarcS OR limitS OR gapS OR sparsS OR circumscribS) adj2 
(data OR evidence)).tw. 

Trial adj2 (evidence OR data) adjS (unavailable OR available OR exist) 

(('phase ir or 'phase 1' or 'phase 1' or 'phase 2') adj2 trialS).tw.(phase adj 
('1' or T or '11' or '2' or 'Ha' or '2a')).tw. 

(evidences) adj2 developmentS).tw. 

('research and development' OR 'R&D').tw. 

((earlS or midS or develops or formativeS or determinatives or decisivS or 
designs or concepts or investigations) adj2 (phase? or processS or stage? 
or 'life cycle' or 'lifecycle' or cycle?)).tw. 

((emerging OR earlS OR pilot OR preSdevelopS OR newS OR novel OR 
nascent OR original OR groundbreaking OR 'ground-breaking' OR pioneerS 
OR 'cutting-edge' OR 'cutting edge' OR 'leading-edge' OR 'leading edge' OR 
radicals OR trailblazS OR trendsetS OR seminalS OR innovatiS or 
unconventionS OR innovateS OR emergent OR constructive OR promising 
OR developing OR 'recently introduced') adjS (technologS OR drug? OR 
treatment? OR intervention? OR pharmaceutical? OR device? OR diagnosS 
OR screens OR therapeuticS OR implements OR development? OR service)).tw. 
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likely cost-effectiveness of the technology. The evalu- 
ation is to provide decision support for product and pol- 
icy development. There is limited supporting evidence 
regarding the effectiveness of the technology. Lastly, 
early evaluation may have an influence on the design 
and future performance of the technology. 

The search strategy has been developed by an informa- 
tion analyst with the review team and will be peer- 
reviewed by another information analyst from a systematic 
review research group [23]. Peer-reviewing of the litera- 
ture search strategy will be conducted according to the 
Peer Review of Electronic Search Strategies (PRESS) 
statement [24]. The protocol for this systematic review 
is registered at PROSPERO, the International prospect- 
ive register of systematic reviews (registration number 
CRD42013004636) [25]. 

Additional citations will be identified via scanning the 
reference lists, author searching of potentially relevant 
studies, and forward citation searching in Scopus. We 
suspect that many early economic evaluation studies are 
unpublished, especially those conducted by industry for 
internal use only. There appears to be no straightforward 
way to identify authors of these unpublished studies. We 
will therefore consider a chain-referral sampling ap- 
proach to sample 'hidden populations' in our attempt to 
contact these authors [26]. 

In the chain-referral sampling approach, a hidden 
population is one in which a sampling frame (i.e., a Ust 
of all the members of the population) cannot be con- 
structed in advance. As an alternative, new members 
are selected from the social network of existing mem- 
bers of the sample. In this approach, first a number of 
seeds are selected [26]. These seeds are members of 
the hidden population that have been identified: au- 
thors of published early economic evaluation studies 
who work for industry, and similar individuals identi- 
fied from our contact files. The seeds are contacted 
and form the first stage (stage 0) of the sampling process. 
The seeds identify other members of the population. The 
members identified in the next stage (stage 1) are 
approached and then asked to identify other members. 
This process is continued until no new members could 
be identified. 

We fully recognize that the above sampling scheme may 
not yield a comprehensive collection of unpublished study 
reports. The key barrier to access to internal reports of 
early evaluation may be the reluctance to share proprietary 
information. Possible authors will be contacted by email. 
We will follow up by phone with authors who contributed 
to unpublished studies in order to discuss their involve- 
ment and, if possible, request information about the un- 
published material. For this approach, non-responders are 
defined as a lack of response after three attempts to con- 
tact a member of the sampling population. 



Study selection 

Due to the broad inclusion criteria, screening of poten- 
tially relevant citations will be conducted by pairs of re- 
viewers who are formally trained and experienced in 
economic evaluation. Full-text reports of citations that 
are deemed relevant by one screener will be obtained. 
Reasons for exclusion will be captured for studies that 
are deemed to be not relevant for the review. Full-text 
review will be conducted in pairs; disagreement will be 
discussed and, if necessary, adjudicated by another 
health economist. 

Data abstraction 

Table 2 describes data elements that are going to be ex- 
tracted from the included studies. First, understanding 
the decision context is necessary to discern the contribu- 
tion of early evaluation to decision-making in product 
and policy development. Data pertaining to the decision 
context will be abstracted (e.g., who initiates the evalu- 
ation, who commissions it, the funding sources, why an 
evaluation is needed right now, who conducts the evalu- 
ation, and the primary target audience). Other abstracted 
data include the decision problem, disease classification, 
and target population, technology type (e.g., pharmaceuti- 
cals, medical devices, biomarkers), and technology stage 
(e.g., pre-clinical, phase I or II studies). Data pertaining to 
the methodological aspects of an early evaluation will be 
abstracted, especially how authors of included studies han- 
dled uncertainty in the early evaluation (e.g., probabilistic 
sensitivity analysis, value of information analysis). 

Open text coding will be used to capture the following 
descriptions: the likely effectiveness of a new technology 
and how the associated health consequences and re- 
sources used were measured and valued. These descrip- 
tions will be used to discern the data sources, methods, 
and assumptions involved in the early evaluation. We 
will assess as to whether the results of an included study 
are adequate to answer the main study questions or ob- 
jective, according to authors of included studies and ac- 
cording to the reviewers. A pair of reviewers using the 
Dedoose software for qualitative literature review will 
abstract each included study independently [31]. 

Evidence synthesis 

Characteristics of early economic evaluation studies will 
be summarized according to the various aspects that de- 
fine early evaluation (Table 2). Given the qualitative na- 
ture of this systematic review, there will be no quantitative 
evidence synthesis. Unpublished studies obtained from 
the chain-referral sampling will be summarized separately 
because the representativeness of this sample of unpub- 
lished studies (relative to the population of early economic 
evaluation conducted to support internal decisions) is 
uncertain. 
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Table 2 Abstracted data from included studies 



Decision context 
Who initiate the evaluation? 

Who commissioned it? 

What are the funding sources of the study? 

Why is an evaluation needed right now? 

Who conducted the study? 

Primary target audience according to the 
decision context 

Defining the decision problem 

Decision problem 

Disease or clinical problem 

Target population 
Health technology 

Technology type 

Technology stage 
Methodological aspects of the EE 

Perspective 

Type of evaluations 
Basis of the evaluation 
Time horizon 

Adjusted for differential timing? 
Likely effect of the new technology 

Quantification of the likely effect"^ 
Valuing health outcomes 

Valuation of the likely effect^ 
Resources used and costs 

Quantification of the likely impact on resources'^ 
Modeling, if appropriate 

Model type 

Handling of variability and uncertainty 
Scenario analysis (e.g., structural uncertainty) 
Sensitivity analysis (e.g., one-way, multiple-way) 
Probabilistic sensitivity analysis 
Value-of-information analysis 

Presenting results of the economic evaluation 

According to the authors, are results likely to be 
influential to the decision problem? 

According to the reviewers, are results likely to 
be influential to the decision problem? 



Values Source 

For example, policy decision makers, industry^ third-party payers, hospital managers/ [27] 
administrators, clinicians, patients or patient advocacy groups, not reported 

For example, policy decision makers, industry^ third-party payers, hospital managers/ [27] 
administrators, clinicians, patients or patient advocacy groups, not reported 

For example, public, private, both, others, not reported [27] 

Informing product development, informing policies, not reported [27] 

For example, academia, public, private, multiple sources, others, not reported [27] 

For example, policy decision makers, industry^ third-party payers, hospital managers/ [27] 
administrators, clinicians, patients or patient advocacy groups, not reported 

Open coding. Decision problems will be classifled. For example, the primary reason for [28,29] 
initiating the early evaluation could be product development, health policy 

ICD-10 major categories (see listing in footnotes) [30] 

For example, patients, at-risk individuals, general public, unclear [29] 

For example, pharmaceuticals, medical devices, diagnostic devices, other therapeutic technologies [28] 

For example, basic, translational research, clinical research^ [8] 

For example, societal, publicly funded health-care system, third-party payers [28,29! 
(e.g., insurance companies), not reported 

For example, CEA, CUA, CMA, CBA, others [28,29; 

For example, trial-based, model-based, prospective or retrospective analysis, others [29] 

Numerical number, not reported [28,29! 

Numerical percentage, not reported [28,29; 

Open coding to identify data sources, methods, assumptions^ [28,29; 

Open coding to identify data sources, methods, assumptions'^ [28,29; 

Open coding to identify data sources, methods, assumptions^ [28,29; 



For example, decision tree, state transition (e.g., Markov), discrete event simulation, [28,29; 
dynamic transition model, others 



Yes, no [28,29; 

Yes, no [28,29; 

Yes, no [28,29; 

Yes, no [28,29; 

Yes, no, unclear [28,29; 

Yes, no, unclear [28,29; 



^Definition of treatment effect of a technology can be summarized as the difference between the duration and state of health or HRQL (including the impact of any adverse 
effects of treatment) that would be experienced on average by patients receiving the technology and that experienced by the same group were they to receive alternative care; 
"^industry would include pharmaceutical, device, or genomic manufacturers; investors or inventors;^echnologies are ready for diffusion after regulatory approval, certification of 
laboratory quality, or marketing authorization; '^text coding and review of text descriptions across included studies to derive a broad classification for data sources (e.g., expert 
opinion, individual patient data, administrative databases, clinical registries), methods used (e.g., systematic review, meta-analysis, RCT, non-RCQ and types of assumptions 
(e.g., based upon basic science, early clinical experience). CEA cost-effectiveness analysis, CUA cost-utility analysis, CMA cost-minimization analysis, CBA cost-benefit analysis, QALY 
quality-adjusted life years, HRQOL health-related quality of life, ICER incremental cost-effectiveness ratio, CE cost-effectiveness, PSA probabilistic sensitivity analysis. 
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Discussion 

This protocol describes a systematic review of published 
and unpublished studies of early economic evaluation of 
regulated health technologies, when the supporting evi- 
dence for their effectiveness is not fully established. The 
aim is to understand current practice regarding the use 
of early economic evaluation to inform decisions in the 
development of emerging technologies, including deci- 
sions regarding the allocation of private and public re- 
sources for the development of these technologies and 
decisions regarding the planning and design of research 
programs to collect additional evidence to support regu- 
latory and reimbursement requirements. This systematic 
review will summarize the methods used in early eco- 
nomic evaluation and provide data regarding the contri- 
bution of early economic evaluation to decision-making 
for product and policy development. 

Our systematic review is timely. Conceptual discussion 
of early HTA over a decade ago has evolved into con- 
crete implementations over recent years [6,21,32-34]. As 
we understand it, early HTA initiatives now exist in the 
UK, the Netherlands, Austria, and Ontario [35,36,21,37]. 
These initiatives facilitate the collaboration between pri- 
vate enterprises, public sectors, and academia in the de- 
velopment of selective technologies that address the 
needs of patients and health systems and, to varied ex- 
tent, investment expectations [35]. Early economic evi- 
dence may facilitate decisions regarding the selection of 
promising technologies and the complex trade-offs be- 
tween public and private investments [36] . 

Economic evaluation is essential for health technology 
assessment. Ijzerman and colleagues review previous 
work and methods of early health technology assessment 
and propose a conceptual framework for early assess- 
ment that acknowledges the different phases in technol- 
ogy development and the specific decision problems 
faced by different decision makers [8]. They review a 
range of tools to identify and evaluate technologies in 
development, most notably early health economic mod- 
eling. They suggest that early HTA is an emerging field 
of research that will likely become more important be- 
cause of the increasingly complex trade-offs between in- 
vestments and access in medical product development. 

The method outlined in this protocol is sometimes re- 
ferred to as a meta-epidemiological investigation. In par- 
ticular, there are well-established precedence regarding 
the use of systematic review to investigate factors affect- 
ing the conduct and reporting of trials, systematic re- 
views, and economic evaluations of health interventions 
[38-45]. By adapting a systematic approach to our inves- 
tigation, our aim is to collect relevant data from a repre- 
sentative sample of all early economic evaluation studies 
and, therefore, reliably discern the contribution of early 
economic evaluation to product and policy development. 
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